{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "# Build a Convolutional Neural Network using Estimators\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r1/tutorials/estimators/cnn.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/r1/tutorials/estimators/cnn.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "> Note: This is an archived TF1 notebook. These are configured\n",
        "to run in TF2's \n",
        "[compatbility mode](https://www.tensorflow.org/guide/migrate)\n",
        "but will run in TF1 as well. To use TF1 in Colab, use the\n",
        "[%tensorflow_version 1.x](https://colab.research.google.com/notebooks/tensorflow_version.ipynb)\n",
        "magic."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "The `tf.layers` module provides a high-level API that makes\n",
        "it easy to construct a neural network. It provides methods that facilitate the\n",
        "creation of dense (fully connected) layers and convolutional layers, adding\n",
        "activation functions, and applying dropout regularization. In this tutorial,\n",
        "you'll learn how to use `layers` to build a convolutional neural network model\n",
        "to recognize the handwritten digits in the MNIST data set.\n",
        "\n",
        "![handwritten digits 0–9 from the MNIST data set](https://www.tensorflow.org/images/mnist_0-9.png)\n",
        "\n",
        "The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) comprises 60,000\n",
        "training examples and 10,000 test examples of the handwritten digits 0–9,\n",
        "formatted as 28x28-pixel monochrome images."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTe-6uXpP2Ts"
      },
      "source": [
        "## Get Started\n",
        "\n",
        "Let's set up the imports for our TensorFlow program:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6-tpguHLP6Rm"
      },
      "outputs": [],
      "source": [
        "import tensorflow.compat.v1 as tf\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.INFO)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4j5yyyDFQgSB"
      },
      "source": [
        "## Intro to Convolutional Neural Networks\n",
        "\n",
        "Convolutional neural networks (CNNs) are the current state-of-the-art model\n",
        "architecture for image classification tasks. CNNs apply a series of filters to\n",
        "the raw pixel data of an image to extract and learn higher-level features, which\n",
        "the model can then use for classification. CNNs contains three components:\n",
        "\n",
        "*   **Convolutional layers**, which apply a specified number of convolution\n",
        "    filters to the image. For each subregion, the layer performs a set of\n",
        "    mathematical operations to produce a single value in the output feature map.\n",
        "    Convolutional layers then typically apply a\n",
        "    [ReLU activation function](https://en.wikipedia.org/wiki/Rectifier_\\(neural_networks\\)) to\n",
        "    the output to introduce nonlinearities into the model.\n",
        "\n",
        "*   **Pooling layers**, which\n",
        "    [downsample the image data](https://en.wikipedia.org/wiki/Convolutional_neural_network#Pooling_layer)\n",
        "    extracted by the convolutional layers to reduce the dimensionality of the\n",
        "    feature map in order to decrease processing time. A commonly used pooling\n",
        "    algorithm is max pooling, which extracts subregions of the feature map\n",
        "    (e.g., 2x2-pixel tiles), keeps their maximum value, and discards all other\n",
        "    values.\n",
        "\n",
        "*   **Dense (fully connected) layers**, which perform classification on the\n",
        "    features extracted by the convolutional layers and downsampled by the\n",
        "    pooling layers. In a dense layer, every node in the layer is connected to\n",
        "    every node in the preceding layer.\n",
        "\n",
        "Typically, a CNN is composed of a stack of convolutional modules that perform\n",
        "feature extraction. Each module consists of a convolutional layer followed by a\n",
        "pooling layer. The last convolutional module is followed by one or more dense\n",
        "layers that perform classification. The final dense layer in a CNN contains a\n",
        "single node for each target class in the model (all the possible classes the\n",
        "model may predict), with a\n",
        "[softmax](https://en.wikipedia.org/wiki/Softmax_function) activation function to\n",
        "generate a value between 0–1 for each node (the sum of all these softmax values\n",
        "is equal to 1). We can interpret the softmax values for a given image as\n",
        "relative measurements of how likely it is that the image falls into each target\n",
        "class.\n",
        "\n",
        "Note: For a more comprehensive walkthrough of CNN architecture, see Stanford University's [Convolutional Neural Networks for Visual Recognition course material](https://cs231n.github.io/convolutional-networks/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j23E_Z0FQvZB"
      },
      "source": [
        "## Building the CNN MNIST Classifier\n",
        "\n",
        "Let's build a model to classify the images in the MNIST dataset using the\n",
        "following CNN architecture:\n",
        "\n",
        "1.  **Convolutional Layer #1**: Applies 32 5x5 filters (extracting 5x5-pixel\n",
        "    subregions), with ReLU activation function\n",
        "2.  **Pooling Layer #1**: Performs max pooling with a 2x2 filter and stride of 2\n",
        "    (which specifies that pooled regions do not overlap)\n",
        "3.  **Convolutional Layer #2**: Applies 64 5x5 filters, with ReLU activation\n",
        "    function\n",
        "4.  **Pooling Layer #2**: Again, performs max pooling with a 2x2 filter and\n",
        "    stride of 2\n",
        "5.  **Dense Layer #1**: 1,024 neurons, with dropout regularization rate of 0.4\n",
        "    (probability of 0.4 that any given element will be dropped during training)\n",
        "6.  **Dense Layer #2 (Logits Layer)**: 10 neurons, one for each digit target\n",
        "    class (0–9).\n",
        "\n",
        "The `tf.layers` module contains methods to create each of the three layer types\n",
        "above:\n",
        "\n",
        "*   `conv2d()`. Constructs a two-dimensional convolutional layer. Takes number\n",
        "    of filters, filter kernel size, padding, and activation function as\n",
        "    arguments.\n",
        "*   `max_pooling2d()`. Constructs a two-dimensional pooling layer using the\n",
        "    max-pooling algorithm. Takes pooling filter size and stride as arguments.\n",
        "*   `dense()`. Constructs a dense layer. Takes number of neurons and activation\n",
        "    function as arguments.\n",
        "\n",
        "Each of these methods accepts a tensor as input and returns a transformed tensor\n",
        "as output. This makes it easy to connect one layer to another: just take the\n",
        "output from one layer-creation method and supply it as input to another.\n",
        "\n",
        "Add the following `cnn_model_fn` function, which\n",
        "conforms to the interface expected by TensorFlow's Estimator API (more on this\n",
        "later in [Create the Estimator](#create-the-estimator)). This function takes\n",
        "MNIST feature data, labels, and mode (from\n",
        "`tf.estimator.ModeKeys`: `TRAIN`, `EVAL`, `PREDICT`) as arguments;\n",
        "configures the CNN; and returns predictions, loss, and a training operation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gMR-_3rkRKPa"
      },
      "outputs": [],
      "source": [
        "def cnn_model_fn(features, labels, mode):\n",
        "  \"\"\"Model function for CNN.\"\"\"\n",
        "  # Input Layer\n",
        "  input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n",
        "\n",
        "  # Convolutional Layer #1\n",
        "  conv1 = tf.layers.conv2d(\n",
        "      inputs=input_layer,\n",
        "      filters=32,\n",
        "      kernel_size=[5, 5],\n",
        "      padding=\"same\",\n",
        "      activation=tf.nn.relu)\n",
        "\n",
        "  # Pooling Layer #1\n",
        "  pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n",
        "\n",
        "  # Convolutional Layer #2 and Pooling Layer #2\n",
        "  conv2 = tf.layers.conv2d(\n",
        "      inputs=pool1,\n",
        "      filters=64,\n",
        "      kernel_size=[5, 5],\n",
        "      padding=\"same\",\n",
        "      activation=tf.nn.relu)\n",
        "  pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n",
        "\n",
        "  # Dense Layer\n",
        "  pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])\n",
        "  dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)\n",
        "  dropout = tf.layers.dropout(\n",
        "      inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
        "\n",
        "  # Logits Layer\n",
        "  logits = tf.layers.dense(inputs=dropout, units=10)\n",
        "\n",
        "  predictions = {\n",
        "      # Generate predictions (for PREDICT and EVAL mode)\n",
        "      \"classes\": tf.argmax(input=logits, axis=1),\n",
        "      # Add `softmax_tensor` to the graph. It is used for PREDICT and by the\n",
        "      # `logging_hook`.\n",
        "      \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
        "  }\n",
        "\n",
        "  if mode == tf.estimator.ModeKeys.PREDICT:\n",
        "    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
        "\n",
        "  # Calculate Loss (for both TRAIN and EVAL modes)\n",
        "  loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)\n",
        "\n",
        "  # Configure the Training Op (for TRAIN mode)\n",
        "  if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
        "    train_op = optimizer.minimize(\n",
        "        loss=loss,\n",
        "        global_step=tf.train.get_global_step())\n",
        "    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)\n",
        "\n",
        "  # Add evaluation metrics (for EVAL mode)\n",
        "  eval_metric_ops = {\n",
        "      \"accuracy\": tf.metrics.accuracy(\n",
        "          labels=labels, predictions=predictions[\"classes\"])\n",
        "  }\n",
        "  return tf.estimator.EstimatorSpec(\n",
        "      mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b7z8qC9FRSLB"
      },
      "source": [
        "The following sections (with headings corresponding to each code block above)\n",
        "dive deeper into the `tf.layers` code used to create each layer, as well as how\n",
        "to calculate loss, configure the training op, and generate predictions. If\n",
        "you're already experienced with CNNs and [TensorFlow `Estimator`s](../../guide/custom_estimators.md),\n",
        "and find the above code intuitive, you may want to skim these sections or just\n",
        "skip ahead to [\"Training and Evaluating the CNN MNIST Classifier\"](#train_eval_mnist)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sFBXEYRlRUWu"
      },
      "source": [
        "### Input Layer\n",
        "\n",
        "The methods in the `layers` module for creating convolutional and pooling layers\n",
        "for two-dimensional image data expect input tensors to have a shape of\n",
        "<code>[<em>batch_size</em>, <em>image_height</em>, <em>image_width</em>,\n",
        "<em>channels</em>]</code> by default. This behavior can be changed using the\n",
        "<code><em>data_format</em></code> parameter; defined as follows:\n",
        "\n",
        "*   `batch_size` —Size of the subset of examples to use when performing\n",
        "    gradient descent during training.\n",
        "*   `image_height` —Height of the example images.\n",
        "*   `image_width` —Width of the example images.\n",
        "*   `channels` —Number of color channels in the example images. For color\n",
        "    images, the number of channels is 3 (red, green, blue). For monochrome\n",
        "    images, there is just 1 channel (black).\n",
        "*   `data_format` —A string, one of `channels_last` (default) or `channels_first`.\n",
        "      `channels_last` corresponds to inputs with shape\n",
        "      `(batch, ..., channels)` while `channels_first` corresponds to\n",
        "      inputs with shape `(batch, channels, ...)`.\n",
        "\n",
        "Here, our MNIST dataset is composed of monochrome 28x28 pixel images, so the\n",
        "desired shape for our input layer is <code>[<em>batch_size</em>, 28, 28,\n",
        "1]</code>.\n",
        "\n",
        "To convert our input feature map (`features`) to this shape, we can perform the\n",
        "following `reshape` operation:\n",
        "\n",
        "```\n",
        "input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n",
        "```\n",
        "\n",
        "Note that we've indicated `-1` for batch size, which specifies that this\n",
        "dimension should be dynamically computed based on the number of input values in\n",
        "`features[\"x\"]`, holding the size of all other dimensions constant. This allows\n",
        "us to treat `batch_size` as a hyperparameter that we can tune. For example, if\n",
        "we feed examples into our model in batches of 5, `features[\"x\"]` will contain\n",
        "3,920 values (one value for each pixel in each image), and `input_layer` will\n",
        "have a shape of `[5, 28, 28, 1]`. Similarly, if we feed examples in batches of\n",
        "100, `features[\"x\"]` will contain 78,400 values, and `input_layer` will have a\n",
        "shape of `[100, 28, 28, 1]`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iU8Jr1_JRiKA"
      },
      "source": [
        "### Convolutional Layer #1\n",
        "\n",
        "In our first convolutional layer, we want to apply 32 5x5 filters to the input\n",
        "layer, with a ReLU activation function. We can use the `conv2d()` method in the\n",
        "`layers` module to create this layer as follows:\n",
        "\n",
        "```\n",
        "conv1 = tf.layers.conv2d(\n",
        "    inputs=input_layer,\n",
        "    filters=32,\n",
        "    kernel_size=[5, 5],\n",
        "    padding=\"same\",\n",
        "    activation=tf.nn.relu)\n",
        "```\n",
        "\n",
        "The `inputs` argument specifies our input tensor, which must have the shape\n",
        "<code>[<em>batch_size</em>, <em>image_height</em>, <em>image_width</em>,\n",
        "<em>channels</em>]</code>. Here, we're connecting our first convolutional layer\n",
        "to `input_layer`, which has the shape <code>[<em>batch_size</em>, 28, 28,\n",
        "1]</code>.\n",
        "\n",
        "Note: `conv2d()` will instead accept a shape of `[<em>batch_size</em>, <em>channels</em>, <em>image_height</em>, <em>image_width</em>]` when passed the argument `data_format=channels_first`.\n",
        "\n",
        "The `filters` argument specifies the number of filters to apply (here, 32), and\n",
        "`kernel_size` specifies the dimensions of the filters as `[<em>height</em>,\n",
        "<em>width</em>]</code> (here, <code>[5, 5]`).\n",
        "\n",
        "<p class=\"tip\"><b>TIP:</b> If filter height and width have the same value, you can instead specify a\n",
        "single integer for <code>kernel_size</code>—e.g., <code>kernel_size=5</code>.</p>\n",
        "\n",
        "The `padding` argument specifies one of two enumerated values\n",
        "(case-insensitive): `valid` (default value) or `same`. To specify that the\n",
        "output tensor should have the same height and width values as the input tensor,\n",
        "we set `padding=same` here, which instructs TensorFlow to add 0 values to the\n",
        "edges of the input tensor to preserve height and width of 28. (Without padding,\n",
        "a 5x5 convolution over a 28x28 tensor will produce a 24x24 tensor, as there are\n",
        "24x24 locations to extract a 5x5 tile from a 28x28 grid.)\n",
        "\n",
        "The `activation` argument specifies the activation function to apply to the\n",
        "output of the convolution. Here, we specify ReLU activation with\n",
        "`tf.nn.relu`.\n",
        "\n",
        "Our output tensor produced by `conv2d()` has a shape of\n",
        "<code>[<em>batch_size</em>, 28, 28, 32]</code>: the same height and width\n",
        "dimensions as the input, but now with 32 channels holding the output from each\n",
        "of the filters."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8qzx1ZMFRqt_"
      },
      "source": [
        "### Pooling Layer #1\n",
        "\n",
        "Next, we connect our first pooling layer to the convolutional layer we just\n",
        "created. We can use the `max_pooling2d()` method in `layers` to construct a\n",
        "layer that performs max pooling with a 2x2 filter and stride of 2:\n",
        "\n",
        "```\n",
        "pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n",
        "```\n",
        "\n",
        "Again, `inputs` specifies the input tensor, with a shape of\n",
        "<code>[<em>batch_size</em>, <em>image_height</em>, <em>image_width</em>,\n",
        "<em>channels</em>]</code>. Here, our input tensor is `conv1`, the output from\n",
        "the first convolutional layer, which has a shape of <code>[<em>batch_size</em>,\n",
        "28, 28, 32]</code>.\n",
        "\n",
        "Note: As with <code>conv2d()</code>, <code>max_pooling2d()</code> will instead\n",
        "accept a shape of <code>[<em>batch_size</em>, <em>channels</em>,\n",
        "<em>image_height</em>, <em>image_width</em>]</code> when passed the argument\n",
        "<code>data_format=channels_first</code>.\n",
        "\n",
        "The `pool_size` argument specifies the size of the max pooling filter as\n",
        "<code>[<em>height</em>, <em>width</em>]</code> (here, `[2, 2]`). If both\n",
        "dimensions have the same value, you can instead specify a single integer (e.g.,\n",
        "`pool_size=2`).\n",
        "\n",
        "The `strides` argument specifies the size of the stride. Here, we set a stride\n",
        "of 2, which indicates that the subregions extracted by the filter should be\n",
        "separated by 2 pixels in both the height and width dimensions (for a 2x2 filter,\n",
        "this means that none of the regions extracted will overlap). If you want to set\n",
        "different stride values for height and width, you can instead specify a tuple or\n",
        "list (e.g., `stride=[3, 6]`).\n",
        "\n",
        "Our output tensor produced by `max_pooling2d()` (`pool1`) has a shape of\n",
        "<code>[<em>batch_size</em>, 14, 14, 32]</code>: the 2x2 filter reduces height and width by 50% each."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xXej53NlRzFh"
      },
      "source": [
        "### Convolutional Layer #2 and Pooling Layer #2\n",
        "\n",
        "We can connect a second convolutional and pooling layer to our CNN using\n",
        "`conv2d()` and `max_pooling2d()` as before. For convolutional layer #2, we\n",
        "configure 64 5x5 filters with ReLU activation, and for pooling layer #2, we use\n",
        "the same specs as pooling layer #1 (a 2x2 max pooling filter with stride of 2):\n",
        "\n",
        "```\n",
        "conv2 = tf.layers.conv2d(\n",
        "    inputs=pool1,\n",
        "    filters=64,\n",
        "    kernel_size=[5, 5],\n",
        "    padding=\"same\",\n",
        "    activation=tf.nn.relu)\n",
        "\n",
        "pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n",
        "```\n",
        "\n",
        "Note that convolutional layer #2 takes the output tensor of our first pooling\n",
        "layer (`pool1`) as input, and produces the tensor `conv2` as output. `conv2`\n",
        "has a shape of <code>[<em>batch_size</em>, 14, 14, 64]</code>, the same height and width as `pool1` (due to `padding=\"same\"`), and 64 channels for the 64\n",
        "filters applied.\n",
        "\n",
        "Pooling layer #2 takes `conv2` as input, producing `pool2` as output. `pool2`\n",
        "has shape <code>[<em>batch_size</em>, 7, 7, 64]</code> (50% reduction of height and width from `conv2`)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jjmLqVP7R7z6"
      },
      "source": [
        "### Dense Layer\n",
        "\n",
        "Next, we want to add a dense layer (with 1,024 neurons and ReLU activation) to\n",
        "our CNN to perform classification on the features extracted by the\n",
        "convolution/pooling layers. Before we connect the layer, however, we'll flatten\n",
        "our feature map (`pool2`) to shape <code>[<em>batch_size</em>,\n",
        "<em>features</em>]</code>, so that our tensor has only two dimensions:\n",
        "\n",
        "```\n",
        "pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])\n",
        "```\n",
        "\n",
        "In the `reshape()` operation above, the `-1` signifies that the *`batch_size`*\n",
        "dimension will be dynamically calculated based on the number of examples in our\n",
        "input data. Each example has 7 (`pool2` height) * 7 (`pool2` width) * 64\n",
        "(`pool2` channels) features, so we want the `features` dimension to have a value\n",
        "of 7 * 7 * 64 (3136 in total). The output tensor, `pool2_flat`, has shape\n",
        "<code>[<em>batch_size</em>, 3136]</code>.\n",
        "\n",
        "Now, we can use the `dense()` method in `layers` to connect our dense layer as\n",
        "follows:\n",
        "\n",
        "```\n",
        "dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)\n",
        "```\n",
        "\n",
        "The `inputs` argument specifies the input tensor: our flattened feature map,\n",
        "`pool2_flat`. The `units` argument specifies the number of neurons in the dense\n",
        "layer (1,024). The `activation` argument takes the activation function; again,\n",
        "we'll use `tf.nn.relu` to add ReLU activation.\n",
        "\n",
        "To help improve the results of our model, we also apply dropout regularization\n",
        "to our dense layer, using the `dropout` method in `layers`:\n",
        "\n",
        "```\n",
        "dropout = tf.layers.dropout(\n",
        "    inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)\n",
        "```\n",
        "\n",
        "Again, `inputs` specifies the input tensor, which is the output tensor from our\n",
        "dense layer (`dense`).\n",
        "\n",
        "The `rate` argument specifies the dropout rate; here, we use `0.4`, which means\n",
        "40% of the elements will be randomly dropped out during training.\n",
        "\n",
        "The `training` argument takes a boolean specifying whether or not the model is\n",
        "currently being run in training mode; dropout will only be performed if\n",
        "`training` is `True`. Here, we check if the `mode` passed to our model function\n",
        "`cnn_model_fn` is `TRAIN` mode.\n",
        "\n",
        "Our output tensor `dropout` has shape <code>[<em>batch_size</em>, 1024]</code>."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzUcwkCZSTF7"
      },
      "source": [
        "### Logits Layer\n",
        "\n",
        "The final layer in our neural network is the logits layer, which will return the\n",
        "raw values for our predictions. We create a dense layer with 10 neurons (one for\n",
        "each target class 0–9), with linear activation (the default):\n",
        "\n",
        "```\n",
        "logits = tf.layers.dense(inputs=dropout, units=10)\n",
        "```\n",
        "\n",
        "Our final output tensor of the CNN, `logits`, has shape `[batch_size, 10]`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y3uJ0V1KSakc"
      },
      "source": [
        "### Generate Predictions {#generate_predictions}\n",
        "\n",
        "The logits layer of our model returns our predictions as raw values in a\n",
        "<code>[<em>batch_size</em>, 10]</code>-dimensional tensor. Let's convert these\n",
        "raw values into two different formats that our model function can return:\n",
        "\n",
        "*   The **predicted class** for each example: a digit from 0–9.\n",
        "*   The **probabilities** for each possible target class for each example: the\n",
        "    probability that the example is a 0, is a 1, is a 2, etc.\n",
        "\n",
        "For a given example, our predicted class is the element in the corresponding row\n",
        "of the logits tensor with the highest raw value. We can find the index of this\n",
        "element using the `tf.argmax`\n",
        "function:\n",
        "\n",
        "```\n",
        "tf.argmax(input=logits, axis=1)\n",
        "```\n",
        "\n",
        "The `input` argument specifies the tensor from which to extract maximum\n",
        "values—here `logits`. The `axis` argument specifies the axis of the `input`\n",
        "tensor along which to find the greatest value. Here, we want to find the largest\n",
        "value along the dimension with index of 1, which corresponds to our predictions\n",
        "(recall that our logits tensor has shape <code>[<em>batch_size</em>,\n",
        "10]</code>).\n",
        "\n",
        "We can derive probabilities from our logits layer by applying softmax activation\n",
        "using `tf.nn.softmax`:\n",
        "\n",
        "```\n",
        "tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
        "```\n",
        "\n",
        "Note: We use the `name` argument to explicitly name this operation `softmax_tensor`, so we can reference it later. (We'll set up logging for the softmax values in [\"Set Up a Logging Hook\"](#set-up-a-logging-hook)).\n",
        "\n",
        "We compile our predictions in a dict, and return an `EstimatorSpec` object:\n",
        "\n",
        "```\n",
        "predictions = {\n",
        "    \"classes\": tf.argmax(input=logits, axis=1),\n",
        "    \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n",
        "}\n",
        "if mode == tf.estimator.ModeKeys.PREDICT:\n",
        "  return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2ks_tqSSucg"
      },
      "source": [
        "### Calculate Loss {#calculating-loss}\n",
        "\n",
        "For both training and evaluation, we need to define a\n",
        "[loss function](https://en.wikipedia.org/wiki/Loss_function)\n",
        "that measures how closely the model's predictions match the target classes. For\n",
        "multiclass classification problems like MNIST,\n",
        "[cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) is typically used\n",
        "as the loss metric. The following code calculates cross entropy when the model\n",
        "runs in either `TRAIN` or `EVAL` mode:\n",
        "\n",
        "```\n",
        "loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)\n",
        "```\n",
        "\n",
        "Let's take a closer look at what's happening above.\n",
        "\n",
        "Our `labels` tensor contains a list of prediction indices for our examples, e.g. `[1,\n",
        "9, ...]`. `logits` contains the linear outputs of our last layer.\n",
        "\n",
        "`tf.losses.sparse_softmax_cross_entropy`, calculates the softmax crossentropy\n",
        "(aka: categorical crossentropy, negative log-likelihood) from these two inputs\n",
        "in an efficient, numerically stable way."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YgE7Ll3pS2FG"
      },
      "source": [
        "### Configure the Training Op\n",
        "\n",
        "In the previous section, we defined loss for our CNN as the softmax\n",
        "cross-entropy of the logits layer and our labels. Let's configure our model to\n",
        "optimize this loss value during training. We'll use a learning rate of 0.001 and\n",
        "[stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)\n",
        "as the optimization algorithm:\n",
        "\n",
        "```\n",
        "if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "  optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
        "  train_op = optimizer.minimize(\n",
        "      loss=loss,\n",
        "      global_step=tf.train.get_global_step())\n",
        "  return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEJPnXAzS6m9"
      },
      "source": [
        "Note: For a more in-depth look at configuring training ops for Estimator model functions, see [\"Defining the training op for the model\"](../../guide/custom_estimators.md#defining-the-training-op-for-the-model) in the [\"Creating Estimations in tf.estimator\"](../../guide/custom_estimators.md) tutorial."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQuGDWvHTAib"
      },
      "source": [
        "### Add evaluation metrics\n",
        "\n",
        "To add accuracy metric in our model, we define `eval_metric_ops` dict in EVAL\n",
        "mode as follows:\n",
        "\n",
        "```\n",
        "eval_metric_ops = {\n",
        "    \"accuracy\": tf.metrics.accuracy(\n",
        "        labels=labels, predictions=predictions[\"classes\"])\n",
        "}\n",
        "return tf.estimator.EstimatorSpec(\n",
        "    mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y2Bwe-AdTRzX"
      },
      "source": [
        "<a id=\"train_eval_mnist\"></a>\n",
        "## Training and Evaluating the CNN MNIST Classifier\n",
        "\n",
        "We've coded our MNIST CNN model function; now we're ready to train and evaluate\n",
        "it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6EC9aOY2TTLU"
      },
      "source": [
        "### Load Training and Test Data\n",
        "\n",
        "First, let's load our training and test data with the following code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ccobb0qETV-S"
      },
      "outputs": [],
      "source": [
        "# Load training and eval data\n",
        "((train_data, train_labels),\n",
        " (eval_data, eval_labels)) = tf.keras.datasets.mnist.load_data()\n",
        "\n",
        "train_data = train_data/np.float32(255)\n",
        "train_labels = train_labels.astype(np.int32)  # not required\n",
        "\n",
        "eval_data = eval_data/np.float32(255)\n",
        "eval_labels = eval_labels.astype(np.int32)  # not required"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8l84-IxSTZnO"
      },
      "source": [
        "We store the training feature data (the raw pixel values for 55,000 images of\n",
        "hand-drawn digits) and training labels (the corresponding value from 0–9 for\n",
        "each image) as [numpy\n",
        "arrays](https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html)\n",
        "in `train_data` and `train_labels`, respectively. Similarly, we store the\n",
        "evaluation feature data (10,000 images) and evaluation labels in `eval_data`\n",
        "and `eval_labels`, respectively."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2_Isc7kTa45"
      },
      "source": [
        "### Create the Estimator {#create-the-estimator}\n",
        "\n",
        "Next, let's create an `Estimator` (a TensorFlow class for performing high-level\n",
        "model training, evaluation, and inference) for our model. Add the following code\n",
        "to `main()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yjC6HdwZTdg4"
      },
      "outputs": [],
      "source": [
        "# Create the Estimator\n",
        "mnist_classifier = tf.estimator.Estimator(\n",
        "    model_fn=cnn_model_fn, model_dir=\"/tmp/mnist_convnet_model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f78EBcg7TfTU"
      },
      "source": [
        "The `model_fn` argument specifies the model function to use for training,\n",
        "evaluation, and prediction; we pass it the `cnn_model_fn` we created in\n",
        "[\"Building the CNN MNIST Classifier.\"](#building-the-cnn-mnist-classifier) The\n",
        "`model_dir` argument specifies the directory where model data (checkpoints) will\n",
        "be saved (here, we specify the temp directory `/tmp/mnist_convnet_model`, but\n",
        "feel free to change to another directory of your choice).\n",
        "\n",
        "Note: For an in-depth walkthrough of the TensorFlow `Estimator` API, see the tutorial [Creating Estimators in tf.estimator](../../guide/custom_estimators.md)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_6ow7hVYTm3f"
      },
      "source": [
        "### Set Up a Logging Hook {#set_up_a_logging_hook}\n",
        "\n",
        "Since CNNs can take a while to train, let's set up some logging so we can track\n",
        "progress during training. We can use TensorFlow's `tf.train.SessionRunHook` to create a\n",
        "`tf.train.LoggingTensorHook`\n",
        "that will log the probability values from the softmax layer of our CNN. Add the\n",
        "following to `main()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S6T10kssTpdz"
      },
      "outputs": [],
      "source": [
        "# Set up logging for predictions\n",
        "tensors_to_log = {\"probabilities\": \"softmax_tensor\"}\n",
        "\n",
        "logging_hook = tf.train.LoggingTensorHook(\n",
        "    tensors=tensors_to_log, every_n_iter=50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZdtZ6JQTsmg"
      },
      "source": [
        "We store a dict of the tensors we want to log in `tensors_to_log`. Each key is a\n",
        "label of our choice that will be printed in the log output, and the\n",
        "corresponding label is the name of a `Tensor` in the TensorFlow graph. Here, our\n",
        "`probabilities` can be found in `softmax_tensor`, the name we gave our softmax\n",
        "operation earlier when we generated the probabilities in `cnn_model_fn`.\n",
        "\n",
        "Note: If you don't explicitly assign a name to an operation via the `name` argument, TensorFlow will assign a default name. A couple easy ways to discover the names applied to operations are to visualize your graph on [TensorBoard](../../guide/graph_viz.md)) or to enable the [TensorFlow Debugger (tfdbg)](../../guide/debugger.md).\n",
        "\n",
        "Next, we create the `LoggingTensorHook`, passing `tensors_to_log` to the\n",
        "`tensors` argument. We set `every_n_iter=50`, which specifies that probabilities\n",
        "should be logged after every 50 steps of training."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brVs1dRMT0NM"
      },
      "source": [
        "### Train the Model\n",
        "\n",
        "Now we're ready to train our model, which we can do by creating `train_input_fn`\n",
        "and calling `train()` on `mnist_classifier`. In the `numpy_input_fn` call, we pass the training feature data and labels to\n",
        "`x` (as a dict) and `y`, respectively. We set a `batch_size` of `100` (which\n",
        "means that the model will train on minibatches of 100 examples at each step).\n",
        "`num_epochs=None` means that the model will train until the specified number of\n",
        "steps is reached. We also set `shuffle=True` to shuffle the training data. Then train the model a single step and log the output:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h-dewpleT2sk"
      },
      "outputs": [],
      "source": [
        "# Train the model\n",
        "train_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
        "    x={\"x\": train_data},\n",
        "    y=train_labels,\n",
        "    batch_size=100,\n",
        "    num_epochs=None,\n",
        "    shuffle=True)\n",
        "\n",
        "# train one step and display the probabilties\n",
        "mnist_classifier.train(\n",
        "    input_fn=train_input_fn,\n",
        "    steps=1,\n",
        "    hooks=[logging_hook])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gyNSE3e-14Lq"
      },
      "source": [
        "Now—without logging each step—set `steps=1000` to train the model longer, but in a reasonable time to run this example. Training CNNs is computationally intensive. To increase the accuracy of your model, increase the number of `steps` passed to `train()`, like 20,000 steps."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cri6zqcf2IXY"
      },
      "outputs": [],
      "source": [
        "mnist_classifier.train(input_fn=train_input_fn, steps=1000)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4bQdkLMeUE5U"
      },
      "source": [
        "### Evaluate the Model\n",
        "\n",
        "Once training is complete, we want to evaluate our model to determine its\n",
        "accuracy on the MNIST test set. We call the `evaluate` method, which evaluates\n",
        "the metrics we specified in `eval_metric_ops` argument in the `model_fn`.\n",
        "Add the following to `main()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I0RGiqd0UF0N"
      },
      "outputs": [],
      "source": [
        "eval_input_fn = tf.estimator.inputs.numpy_input_fn(\n",
        "    x={\"x\": eval_data},\n",
        "    y=eval_labels,\n",
        "    num_epochs=1,\n",
        "    shuffle=False)\n",
        "\n",
        "eval_results = mnist_classifier.evaluate(input_fn=eval_input_fn)\n",
        "print(eval_results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JIBVID6dUIXT"
      },
      "source": [
        "To create `eval_input_fn`, we set `num_epochs=1`, so that the model evaluates\n",
        "the metrics over one epoch of data and returns the result. We also set\n",
        "`shuffle=False` to iterate through the data sequentially."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "htmLZ-zEUZZk"
      },
      "source": [
        "## Additional Resources\n",
        "\n",
        "To learn more about TensorFlow Estimators and CNNs in TensorFlow, see the\n",
        "following resources:\n",
        "\n",
        "*   [Creating Estimators in tf.estimator](../../guide/custom_estimators.md)\n",
        "    provides an introduction to the TensorFlow Estimator API. It walks through\n",
        "    configuring an Estimator, writing a model function, calculating loss, and\n",
        "    defining a training op.\n",
        "*   [Advanced Convolutional Neural Networks](../../tutorials/images/deep_cnn.md) walks through how to build a MNIST CNN classification model\n",
        "    *without estimators* using lower-level TensorFlow operations."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "Tce3stUlHN0L"
      ],
      "name": "cnn.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
